


<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">


<html xmlns="http://www.w3.org/1999/xhtml">
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
    
    <title>Fault Tolerance (Scala) &mdash; Akka Documentation</title>
    
    <link rel="stylesheet" href="../_static/style.css" type="text/css" />
    <link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
    <link rel="stylesheet" href="../_static/prettify.css" type="text/css" />
    <link rel="stylesheet" href="../_static/base.css" type="text/css" />
    <link rel="stylesheet" href="../_static/docs.css" type="text/css" />
    <link rel="stylesheet" href="http://fonts.googleapis.com/css?family=Exo:300,400,600,700" type="text/css" />
    
    <script type="text/javascript">
      var DOCUMENTATION_OPTIONS = {
        URL_ROOT:    '../',
        VERSION:     '2.0.1',
        COLLAPSE_INDEX: false,
        FILE_SUFFIX: '.html',
        HAS_SOURCE:  true
      };
    </script>
    <script type="text/javascript" src="../_static/jquery.js"></script>
    <script type="text/javascript" src="../_static/underscore.js"></script>
    <script type="text/javascript" src="../_static/doctools.js"></script>
    <script type="text/javascript" src="../_static/toc.js"></script>
    <script type="text/javascript" src="../_static/prettify.js"></script>
    <script type="text/javascript" src="../_static/highlightCode.js"></script>
    <script type="text/javascript" src="../_static/effects.core.js"></script>
    <script type="text/javascript" src="../_static/effects.highlight.js"></script>
    <script type="text/javascript" src="../_static/scrollTo.js"></script>
    <script type="text/javascript" src="../_static/contentsFix.js"></script>
    <link rel="shortcut icon" href="../_static/favicon.ico"/>
    <link rel="top" title="Akka Documentation" href="../index.html" />
    <link rel="up" title="Scala API" href="index.html" />
    <link rel="next" title="Diagrams of the Fault Tolerance Sample (Scala)" href="fault-tolerance-sample.html" />
    <link rel="prev" title="Dataflow Concurrency (Scala)" href="dataflow.html" /> 
  </head>
  <body>
  <div class="navbar">
    <div class="navbar-inner">
      <div class="container">
        <div class="navbar-logo">
          <a href="http://akka.io"><img src="../_static/logo-small.png" /></a>
        </div>    
        <ul class="nav">
          <li><a href="http://akka.io/docs">Documentation</a></li>
          <li><a href="http://akka.io/downloads">Download</a></li>
          <li><a href="http://groups.google.com/group/akka-user">Mailing List</a></li>
          <li><a href="http://github.com/akka/akka">Code</a></li>           
          <li><a href="http://typesafe.com/products/typesafe-subscription">Commerical Support</a></li>
        </ul>
      </div>
    </div>
  </div>
  <div class="main">
    <div class="container">
      <div class="page-title">Fault Tolerance (Scala)</div><div class="pdf-link"><a href="http://akka.io/docs/akka/2.0.1/Akka.pdf"><img src="../_static/pdf-icon.png" style="height: 40px;" /></a></div></div>
    <div class="main-container">
      <div class="container">
        <div class="row">
          <div class="span12">
            <ul class="breadcrumb">           
              <li>
                 <span class="divider">|</span> <a href="fault-tolerance-sample.html">Diagrams of the Fault Tolerance Sample (Scala)</a> <span class="divider">»</span>
              </li>
              <li>
                <a href="../index.html">Contents</a>
              </li>
              <li>
                <span class="divider">«</span> <a href="dataflow.html">Dataflow Concurrency (Scala)</a> <span class="divider">|</span>
              </li>
              <li>
                Version 2.0.1
              </li>
            </ul>         
          </div>
        </div>
        <div class="row">
          <div class="span9">
            
  <div class="section" id="fault-tolerance-scala">
<span id="id1"></span><h1>Fault Tolerance (Scala)</h1>
<p>As explained in <a class="reference internal" href="../general/actor-systems.html#actor-systems"><em>Actor Systems</em></a> each actor is the supervisor of its
children, and as such each actor defines fault handling supervisor strategy.
This strategy cannot be changed afterwards as it is an integral part of the
actor system’s structure.</p>
<div class="section" id="fault-handling-in-practice">
<h2>Fault Handling in Practice</h2>
<p>First, let us look at a sample that illustrates one way to handle data store errors,
which is a typical source of failure in real world applications. Of course it depends
on the actual application what is possible to do when the data store is unavailable,
but in this sample we use a best effort re-connect approach.</p>
<p>Read the following source code. The inlined comments explain the different pieces of
the fault handling and why they are added. It is also highly recommended to run this
sample as it is easy to follow the log output to understand what is happening in runtime.</p>
<div class="toctree-wrapper compound">
<ul>
<li class="toctree-l1"><a class="reference internal" href="fault-tolerance-sample.html">Diagrams of the Fault Tolerance Sample (Scala)</a></li>
<li class="toctree-l1"><a class="reference internal" href="fault-tolerance-sample.html#full-source-code-of-the-fault-tolerance-sample-scala">Full Source Code of the Fault Tolerance Sample (Scala)</a></li>
</ul>
</div>
<div class="highlight-scala"><div class="highlight"><pre><span class="c1">// imports ...</span>

<span class="cm">/**</span>
<span class="cm"> * Runs the sample</span>
<span class="cm"> */</span>
<span class="k">object</span> <span class="nc">FaultHandlingDocSample</span> <span class="k">extends</span> <span class="nc">App</span> <span class="o">{</span>
  <span class="k">import</span> <span class="nn">Worker._</span>

  <span class="k">val</span> <span class="n">config</span> <span class="k">=</span> <span class="nc">ConfigFactory</span><span class="o">.</span><span class="n">parseString</span><span class="o">(</span><span class="s">&quot;&quot;&quot;</span>
<span class="s">    akka.loglevel = DEBUG</span>
<span class="s">    akka.actor.debug {</span>
<span class="s">      receive = on</span>
<span class="s">      lifecycle = on</span>
<span class="s">    }</span>
<span class="s">    &quot;&quot;&quot;</span><span class="o">)</span>

  <span class="k">val</span> <span class="n">system</span> <span class="k">=</span> <span class="nc">ActorSystem</span><span class="o">(</span><span class="s">&quot;FaultToleranceSample&quot;</span><span class="o">,</span> <span class="n">config</span><span class="o">)</span>
  <span class="k">val</span> <span class="n">worker</span> <span class="k">=</span> <span class="n">system</span><span class="o">.</span><span class="n">actorOf</span><span class="o">(</span><span class="nc">Props</span><span class="o">[</span><span class="kt">Worker</span><span class="o">],</span> <span class="n">name</span> <span class="k">=</span> <span class="s">&quot;worker&quot;</span><span class="o">)</span>
  <span class="k">val</span> <span class="n">listener</span> <span class="k">=</span> <span class="n">system</span><span class="o">.</span><span class="n">actorOf</span><span class="o">(</span><span class="nc">Props</span><span class="o">[</span><span class="kt">Listener</span><span class="o">],</span> <span class="n">name</span> <span class="k">=</span> <span class="s">&quot;listener&quot;</span><span class="o">)</span>
  <span class="c1">// start the work and listen on progress</span>
  <span class="c1">// note that the listener is used as sender of the tell,</span>
  <span class="c1">// i.e. it will receive replies from the worker</span>
  <span class="n">worker</span><span class="o">.</span><span class="n">tell</span><span class="o">(</span><span class="nc">Start</span><span class="o">,</span> <span class="n">sender</span> <span class="k">=</span> <span class="n">listener</span><span class="o">)</span>
<span class="o">}</span>

<span class="cm">/**</span>
<span class="cm"> * Listens on progress from the worker and shuts down the system when enough</span>
<span class="cm"> * work has been done.</span>
<span class="cm"> */</span>
<span class="k">class</span> <span class="nc">Listener</span> <span class="k">extends</span> <span class="nc">Actor</span> <span class="k">with</span> <span class="nc">ActorLogging</span> <span class="o">{</span>
  <span class="k">import</span> <span class="nn">Worker._</span>
  <span class="c1">// If we don&#39;t get any progress within 15 seconds then the service is unavailable</span>
  <span class="n">context</span><span class="o">.</span><span class="n">setReceiveTimeout</span><span class="o">(</span><span class="mi">15</span> <span class="n">seconds</span><span class="o">)</span>

  <span class="k">def</span> <span class="n">receive</span> <span class="k">=</span> <span class="o">{</span>
    <span class="k">case</span> <span class="nc">Progress</span><span class="o">(</span><span class="n">percent</span><span class="o">)</span> <span class="k">⇒</span>
      <span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="o">(</span><span class="s">&quot;Current progress: {} %&quot;</span><span class="o">,</span> <span class="n">percent</span><span class="o">)</span>
      <span class="k">if</span> <span class="o">(</span><span class="n">percent</span> <span class="o">&gt;=</span> <span class="mf">100.0</span><span class="o">)</span> <span class="o">{</span>
        <span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="o">(</span><span class="s">&quot;That&#39;s all, shutting down&quot;</span><span class="o">)</span>
        <span class="n">context</span><span class="o">.</span><span class="n">system</span><span class="o">.</span><span class="n">shutdown</span><span class="o">()</span>
      <span class="o">}</span>

    <span class="k">case</span> <span class="nc">ReceiveTimeout</span> <span class="k">⇒</span>
      <span class="c1">// No progress within 15 seconds, ServiceUnavailable</span>
      <span class="n">log</span><span class="o">.</span><span class="n">error</span><span class="o">(</span><span class="s">&quot;Shutting down due to unavailable service&quot;</span><span class="o">)</span>
      <span class="n">context</span><span class="o">.</span><span class="n">system</span><span class="o">.</span><span class="n">shutdown</span><span class="o">()</span>
  <span class="o">}</span>
<span class="o">}</span>

<span class="c1">// messages ...</span>

<span class="cm">/**</span>
<span class="cm"> * Worker performs some work when it receives the `Start` message.</span>
<span class="cm"> * It will continuously notify the sender of the `Start` message</span>
<span class="cm"> * of current ``Progress``. The `Worker` supervise the `CounterService`.</span>
<span class="cm"> */</span>
<span class="k">class</span> <span class="nc">Worker</span> <span class="k">extends</span> <span class="nc">Actor</span> <span class="k">with</span> <span class="nc">ActorLogging</span> <span class="o">{</span>
  <span class="k">import</span> <span class="nn">Worker._</span>
  <span class="k">import</span> <span class="nn">CounterService._</span>
  <span class="k">implicit</span> <span class="k">val</span> <span class="n">askTimeout</span> <span class="k">=</span> <span class="nc">Timeout</span><span class="o">(</span><span class="mi">5</span> <span class="n">seconds</span><span class="o">)</span>

  <span class="c1">// Stop the CounterService child if it throws ServiceUnavailable</span>
  <span class="k">override</span> <span class="k">val</span> <span class="n">supervisorStrategy</span> <span class="k">=</span> <span class="nc">OneForOneStrategy</span><span class="o">()</span> <span class="o">{</span>
    <span class="k">case</span> <span class="n">_:</span> <span class="nc">CounterService</span><span class="o">.</span><span class="nc">ServiceUnavailable</span> <span class="k">⇒</span> <span class="nc">Stop</span>
  <span class="o">}</span>

  <span class="c1">// The sender of the initial Start message will continuously be notified about progress</span>
  <span class="k">var</span> <span class="n">progressListener</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">ActorRef</span><span class="o">]</span> <span class="k">=</span> <span class="nc">None</span>
  <span class="k">val</span> <span class="n">counterService</span> <span class="k">=</span> <span class="n">context</span><span class="o">.</span><span class="n">actorOf</span><span class="o">(</span><span class="nc">Props</span><span class="o">[</span><span class="kt">CounterService</span><span class="o">],</span> <span class="n">name</span> <span class="k">=</span> <span class="s">&quot;counter&quot;</span><span class="o">)</span>
  <span class="k">val</span> <span class="n">totalCount</span> <span class="k">=</span> <span class="mi">51</span>

  <span class="k">def</span> <span class="n">receive</span> <span class="k">=</span> <span class="nc">LoggingReceive</span> <span class="o">{</span>
    <span class="k">case</span> <span class="nc">Start</span> <span class="k">if</span> <span class="n">progressListener</span><span class="o">.</span><span class="n">isEmpty</span> <span class="k">⇒</span>
      <span class="n">progressListener</span> <span class="k">=</span> <span class="nc">Some</span><span class="o">(</span><span class="n">sender</span><span class="o">)</span>
      <span class="n">context</span><span class="o">.</span><span class="n">system</span><span class="o">.</span><span class="n">scheduler</span><span class="o">.</span><span class="n">schedule</span><span class="o">(</span><span class="nc">Duration</span><span class="o">.</span><span class="nc">Zero</span><span class="o">,</span> <span class="mi">1</span> <span class="n">second</span><span class="o">,</span> <span class="n">self</span><span class="o">,</span> <span class="nc">Do</span><span class="o">)</span>

    <span class="k">case</span> <span class="nc">Do</span> <span class="k">⇒</span>
      <span class="n">counterService</span> <span class="o">!</span> <span class="nc">Increment</span><span class="o">(</span><span class="mi">1</span><span class="o">)</span>
      <span class="n">counterService</span> <span class="o">!</span> <span class="nc">Increment</span><span class="o">(</span><span class="mi">1</span><span class="o">)</span>
      <span class="n">counterService</span> <span class="o">!</span> <span class="nc">Increment</span><span class="o">(</span><span class="mi">1</span><span class="o">)</span>

      <span class="c1">// Send current progress to the initial sender</span>
      <span class="n">counterService</span> <span class="o">?</span> <span class="nc">GetCurrentCount</span> <span class="n">map</span> <span class="o">{</span>
        <span class="k">case</span> <span class="nc">CurrentCount</span><span class="o">(</span><span class="n">_</span><span class="o">,</span> <span class="n">count</span><span class="o">)</span> <span class="k">⇒</span> <span class="nc">Progress</span><span class="o">(</span><span class="mf">100.0</span> <span class="o">*</span> <span class="n">count</span> <span class="o">/</span> <span class="n">totalCount</span><span class="o">)</span>
      <span class="o">}</span> <span class="n">pipeTo</span> <span class="n">progressListener</span><span class="o">.</span><span class="n">get</span>
  <span class="o">}</span>
<span class="o">}</span>

<span class="c1">// messages ...</span>

<span class="cm">/**</span>
<span class="cm"> * Adds the value received in `Increment` message to a persistent</span>
<span class="cm"> * counter. Replies with `CurrentCount` when it is asked for `CurrentCount`.</span>
<span class="cm"> * `CounterService` supervise `Storage` and `Counter`.</span>
<span class="cm"> */</span>
<span class="k">class</span> <span class="nc">CounterService</span> <span class="k">extends</span> <span class="nc">Actor</span> <span class="o">{</span>
  <span class="k">import</span> <span class="nn">CounterService._</span>
  <span class="k">import</span> <span class="nn">Counter._</span>
  <span class="k">import</span> <span class="nn">Storage._</span>

  <span class="c1">// Restart the storage child when StorageException is thrown.</span>
  <span class="c1">// After 3 restarts within 5 seconds it will be stopped.</span>
  <span class="k">override</span> <span class="k">val</span> <span class="n">supervisorStrategy</span> <span class="k">=</span> <span class="nc">OneForOneStrategy</span><span class="o">(</span><span class="n">maxNrOfRetries</span> <span class="k">=</span> <span class="mi">3</span><span class="o">,</span> <span class="n">withinTimeRange</span> <span class="k">=</span> <span class="mi">5</span> <span class="n">seconds</span><span class="o">)</span> <span class="o">{</span>
    <span class="k">case</span> <span class="n">_:</span> <span class="nc">Storage</span><span class="o">.</span><span class="nc">StorageException</span> <span class="k">⇒</span> <span class="nc">Restart</span>
  <span class="o">}</span>

  <span class="k">val</span> <span class="n">key</span> <span class="k">=</span> <span class="n">self</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">name</span>
  <span class="k">var</span> <span class="n">storage</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">ActorRef</span><span class="o">]</span> <span class="k">=</span> <span class="nc">None</span>
  <span class="k">var</span> <span class="n">counter</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">ActorRef</span><span class="o">]</span> <span class="k">=</span> <span class="nc">None</span>
  <span class="k">var</span> <span class="n">backlog</span> <span class="k">=</span> <span class="nc">IndexedSeq</span><span class="o">.</span><span class="n">empty</span><span class="o">[(</span><span class="kt">ActorRef</span>, <span class="kt">Any</span><span class="o">)]</span>
  <span class="k">val</span> <span class="nc">MaxBacklog</span> <span class="k">=</span> <span class="mi">10000</span>

  <span class="k">override</span> <span class="k">def</span> <span class="n">preStart</span><span class="o">()</span> <span class="o">{</span>
    <span class="n">initStorage</span><span class="o">()</span>
  <span class="o">}</span>

  <span class="cm">/**</span>
<span class="cm">   * The child storage is restarted in case of failure, but after 3 restarts,</span>
<span class="cm">   * and still failing it will be stopped. Better to back-off than continuously</span>
<span class="cm">   * failing. When it has been stopped we will schedule a Reconnect after a delay.</span>
<span class="cm">   * Watch the child so we receive Terminated message when it has been terminated.</span>
<span class="cm">   */</span>
  <span class="k">def</span> <span class="n">initStorage</span><span class="o">()</span> <span class="o">{</span>
    <span class="n">storage</span> <span class="k">=</span> <span class="nc">Some</span><span class="o">(</span><span class="n">context</span><span class="o">.</span><span class="n">watch</span><span class="o">(</span><span class="n">context</span><span class="o">.</span><span class="n">actorOf</span><span class="o">(</span><span class="nc">Props</span><span class="o">[</span><span class="kt">Storage</span><span class="o">],</span> <span class="n">name</span> <span class="k">=</span> <span class="s">&quot;storage&quot;</span><span class="o">)))</span>
    <span class="c1">// Tell the counter, if any, to use the new storage</span>
    <span class="n">counter</span> <span class="n">foreach</span> <span class="o">{</span> <span class="k">_</span> <span class="o">!</span> <span class="nc">UseStorage</span><span class="o">(</span><span class="n">storage</span><span class="o">)</span> <span class="o">}</span>
    <span class="c1">// We need the initial value to be able to operate</span>
    <span class="n">storage</span><span class="o">.</span><span class="n">get</span> <span class="o">!</span> <span class="nc">Get</span><span class="o">(</span><span class="n">key</span><span class="o">)</span>
  <span class="o">}</span>

  <span class="k">def</span> <span class="n">receive</span> <span class="k">=</span> <span class="nc">LoggingReceive</span> <span class="o">{</span>

    <span class="k">case</span> <span class="nc">Entry</span><span class="o">(</span><span class="n">k</span><span class="o">,</span> <span class="n">v</span><span class="o">)</span> <span class="k">if</span> <span class="n">k</span> <span class="o">==</span> <span class="n">key</span> <span class="o">&amp;&amp;</span> <span class="n">counter</span> <span class="o">==</span> <span class="nc">None</span> <span class="k">⇒</span>
      <span class="c1">// Reply from Storage of the initial value, now we can create the Counter</span>
      <span class="k">val</span> <span class="n">c</span> <span class="k">=</span> <span class="n">context</span><span class="o">.</span><span class="n">actorOf</span><span class="o">(</span><span class="nc">Props</span><span class="o">(</span><span class="k">new</span> <span class="nc">Counter</span><span class="o">(</span><span class="n">key</span><span class="o">,</span> <span class="n">v</span><span class="o">)))</span>
      <span class="n">counter</span> <span class="k">=</span> <span class="nc">Some</span><span class="o">(</span><span class="n">c</span><span class="o">)</span>
      <span class="c1">// Tell the counter to use current storage</span>
      <span class="n">c</span> <span class="o">!</span> <span class="nc">UseStorage</span><span class="o">(</span><span class="n">storage</span><span class="o">)</span>
      <span class="c1">// and send the buffered backlog to the counter</span>
      <span class="k">for</span> <span class="o">((</span><span class="n">replyTo</span><span class="o">,</span> <span class="n">msg</span><span class="o">)</span> <span class="k">←</span> <span class="n">backlog</span><span class="o">)</span> <span class="n">c</span><span class="o">.</span><span class="n">tell</span><span class="o">(</span><span class="n">msg</span><span class="o">,</span> <span class="n">sender</span> <span class="k">=</span> <span class="n">replyTo</span><span class="o">)</span>
      <span class="n">backlog</span> <span class="k">=</span> <span class="nc">IndexedSeq</span><span class="o">.</span><span class="n">empty</span>

    <span class="k">case</span> <span class="n">msg</span> <span class="k">@</span> <span class="nc">Increment</span><span class="o">(</span><span class="n">n</span><span class="o">)</span>    <span class="k">⇒</span> <span class="n">forwardOrPlaceInBacklog</span><span class="o">(</span><span class="n">msg</span><span class="o">)</span>

    <span class="k">case</span> <span class="n">msg</span> <span class="k">@</span> <span class="nc">GetCurrentCount</span> <span class="k">⇒</span> <span class="n">forwardOrPlaceInBacklog</span><span class="o">(</span><span class="n">msg</span><span class="o">)</span>

    <span class="k">case</span> <span class="nc">Terminated</span><span class="o">(</span><span class="n">actorRef</span><span class="o">)</span> <span class="k">if</span> <span class="nc">Some</span><span class="o">(</span><span class="n">actorRef</span><span class="o">)</span> <span class="o">==</span> <span class="n">storage</span> <span class="k">⇒</span>
      <span class="c1">// After 3 restarts the storage child is stopped.</span>
      <span class="c1">// We receive Terminated because we watch the child, see initStorage.</span>
      <span class="n">storage</span> <span class="k">=</span> <span class="nc">None</span>
      <span class="c1">// Tell the counter that there is no storage for the moment</span>
      <span class="n">counter</span> <span class="n">foreach</span> <span class="o">{</span> <span class="k">_</span> <span class="o">!</span> <span class="nc">UseStorage</span><span class="o">(</span><span class="nc">None</span><span class="o">)</span> <span class="o">}</span>
      <span class="c1">// Try to re-establish storage after while</span>
      <span class="n">context</span><span class="o">.</span><span class="n">system</span><span class="o">.</span><span class="n">scheduler</span><span class="o">.</span><span class="n">scheduleOnce</span><span class="o">(</span><span class="mi">10</span> <span class="n">seconds</span><span class="o">,</span> <span class="n">self</span><span class="o">,</span> <span class="nc">Reconnect</span><span class="o">)</span>

    <span class="k">case</span> <span class="nc">Reconnect</span> <span class="k">⇒</span>
      <span class="c1">// Re-establish storage after the scheduled delay</span>
      <span class="n">initStorage</span><span class="o">()</span>
  <span class="o">}</span>

  <span class="k">def</span> <span class="n">forwardOrPlaceInBacklog</span><span class="o">(</span><span class="n">msg</span><span class="k">:</span> <span class="kt">Any</span><span class="o">)</span> <span class="o">{</span>
    <span class="c1">// We need the initial value from storage before we can start delegate to the counter.</span>
    <span class="c1">// Before that we place the messages in a backlog, to be sent to the counter when</span>
    <span class="c1">// it is initialized.</span>
    <span class="n">counter</span> <span class="k">match</span> <span class="o">{</span>
      <span class="k">case</span> <span class="nc">Some</span><span class="o">(</span><span class="n">c</span><span class="o">)</span> <span class="k">⇒</span> <span class="n">c</span> <span class="n">forward</span> <span class="n">msg</span>
      <span class="k">case</span> <span class="nc">None</span> <span class="k">⇒</span>
        <span class="k">if</span> <span class="o">(</span><span class="n">backlog</span><span class="o">.</span><span class="n">size</span> <span class="o">&gt;=</span> <span class="nc">MaxBacklog</span><span class="o">)</span>
          <span class="k">throw</span> <span class="k">new</span> <span class="nc">ServiceUnavailable</span><span class="o">(</span><span class="s">&quot;CounterService not available, lack of initial value&quot;</span><span class="o">)</span>
        <span class="n">backlog</span> <span class="k">=</span> <span class="n">backlog</span> <span class="o">:+</span> <span class="o">(</span><span class="n">sender</span><span class="o">,</span> <span class="n">msg</span><span class="o">)</span>
    <span class="o">}</span>
  <span class="o">}</span>

<span class="o">}</span>

<span class="c1">// messages ...</span>

<span class="cm">/**</span>
<span class="cm"> * The in memory count variable that will send current</span>
<span class="cm"> * value to the `Storage`, if there is any storage</span>
<span class="cm"> * available at the moment.</span>
<span class="cm"> */</span>
<span class="k">class</span> <span class="nc">Counter</span><span class="o">(</span><span class="n">key</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">initialValue</span><span class="k">:</span> <span class="kt">Long</span><span class="o">)</span> <span class="k">extends</span> <span class="nc">Actor</span> <span class="o">{</span>
  <span class="k">import</span> <span class="nn">Counter._</span>
  <span class="k">import</span> <span class="nn">CounterService._</span>
  <span class="k">import</span> <span class="nn">Storage._</span>

  <span class="k">var</span> <span class="n">count</span> <span class="k">=</span> <span class="n">initialValue</span>
  <span class="k">var</span> <span class="n">storage</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">ActorRef</span><span class="o">]</span> <span class="k">=</span> <span class="nc">None</span>

  <span class="k">def</span> <span class="n">receive</span> <span class="k">=</span> <span class="nc">LoggingReceive</span> <span class="o">{</span>
    <span class="k">case</span> <span class="nc">UseStorage</span><span class="o">(</span><span class="n">s</span><span class="o">)</span> <span class="k">⇒</span>
      <span class="n">storage</span> <span class="k">=</span> <span class="n">s</span>
      <span class="n">storeCount</span><span class="o">()</span>

    <span class="k">case</span> <span class="nc">Increment</span><span class="o">(</span><span class="n">n</span><span class="o">)</span> <span class="k">⇒</span>
      <span class="n">count</span> <span class="o">+=</span> <span class="n">n</span>
      <span class="n">storeCount</span><span class="o">()</span>

    <span class="k">case</span> <span class="nc">GetCurrentCount</span> <span class="k">⇒</span>
      <span class="n">sender</span> <span class="o">!</span> <span class="nc">CurrentCount</span><span class="o">(</span><span class="n">key</span><span class="o">,</span> <span class="n">count</span><span class="o">)</span>

  <span class="o">}</span>

  <span class="k">def</span> <span class="n">storeCount</span><span class="o">()</span> <span class="o">{</span>
    <span class="c1">// Delegate dangerous work, to protect our valuable state.</span>
    <span class="c1">// We can continue without storage.</span>
    <span class="n">storage</span> <span class="n">foreach</span> <span class="o">{</span> <span class="k">_</span> <span class="o">!</span> <span class="nc">Store</span><span class="o">(</span><span class="nc">Entry</span><span class="o">(</span><span class="n">key</span><span class="o">,</span> <span class="n">count</span><span class="o">))</span> <span class="o">}</span>
  <span class="o">}</span>

<span class="o">}</span>

<span class="c1">// messages ...</span>

<span class="cm">/**</span>
<span class="cm"> * Saves key/value pairs to persistent storage when receiving `Store` message.</span>
<span class="cm"> * Replies with current value when receiving `Get` message.</span>
<span class="cm"> * Will throw StorageException if the underlying data store is out of order.</span>
<span class="cm"> */</span>
<span class="k">class</span> <span class="nc">Storage</span> <span class="k">extends</span> <span class="nc">Actor</span> <span class="o">{</span>
  <span class="k">import</span> <span class="nn">Storage._</span>

  <span class="k">val</span> <span class="n">db</span> <span class="k">=</span> <span class="nc">DummyDB</span>

  <span class="k">def</span> <span class="n">receive</span> <span class="k">=</span> <span class="nc">LoggingReceive</span> <span class="o">{</span>
    <span class="k">case</span> <span class="nc">Store</span><span class="o">(</span><span class="nc">Entry</span><span class="o">(</span><span class="n">key</span><span class="o">,</span> <span class="n">count</span><span class="o">))</span> <span class="k">⇒</span> <span class="n">db</span><span class="o">.</span><span class="n">save</span><span class="o">(</span><span class="n">key</span><span class="o">,</span> <span class="n">count</span><span class="o">)</span>
    <span class="k">case</span> <span class="nc">Get</span><span class="o">(</span><span class="n">key</span><span class="o">)</span>                 <span class="k">⇒</span> <span class="n">sender</span> <span class="o">!</span> <span class="nc">Entry</span><span class="o">(</span><span class="n">key</span><span class="o">,</span> <span class="n">db</span><span class="o">.</span><span class="n">load</span><span class="o">(</span><span class="n">key</span><span class="o">).</span><span class="n">getOrElse</span><span class="o">(</span><span class="mi">0L</span><span class="o">))</span>
  <span class="o">}</span>
<span class="o">}</span>

<span class="c1">// dummydb ...</span>
</pre></div>
</div>
</div>
<div class="section" id="creating-a-supervisor-strategy">
<h2>Creating a Supervisor Strategy</h2>
<p>The following sections explain the fault handling mechanism and alternatives
in more depth.</p>
<p>For the sake of demonstration let us consider the following strategy:</p>
<div class="highlight-scala"><div class="highlight"><pre><span class="k">import</span> <span class="nn">akka.actor.OneForOneStrategy</span>
<span class="k">import</span> <span class="nn">akka.actor.SupervisorStrategy._</span>
<span class="k">import</span> <span class="nn">akka.util.duration._</span>

<span class="k">override</span> <span class="k">val</span> <span class="n">supervisorStrategy</span> <span class="k">=</span> <span class="nc">OneForOneStrategy</span><span class="o">(</span><span class="n">maxNrOfRetries</span> <span class="k">=</span> <span class="mi">10</span><span class="o">,</span> <span class="n">withinTimeRange</span> <span class="k">=</span> <span class="mi">1</span> <span class="n">minute</span><span class="o">)</span> <span class="o">{</span>
  <span class="k">case</span> <span class="n">_:</span> <span class="nc">ArithmeticException</span>      <span class="k">⇒</span> <span class="nc">Resume</span>
  <span class="k">case</span> <span class="n">_:</span> <span class="nc">NullPointerException</span>     <span class="k">⇒</span> <span class="nc">Restart</span>
  <span class="k">case</span> <span class="n">_:</span> <span class="nc">IllegalArgumentException</span> <span class="k">⇒</span> <span class="nc">Stop</span>
  <span class="k">case</span> <span class="n">_:</span> <span class="nc">Exception</span>                <span class="k">⇒</span> <span class="nc">Escalate</span>
<span class="o">}</span>
</pre></div>
</div>
<p>I have chosen a few well-known exception types in order to demonstrate the
application of the fault handling directives described in <a class="reference internal" href="../general/supervision.html#supervision"><em>Supervision and Monitoring</em></a>.
First off, it is a one-for-one strategy, meaning that each child is treated
separately (an all-for-one strategy works very similarly, the only difference
is that any decision is applied to all children of the supervisor, not only the
failing one). There are limits set on the restart frequency, namely maximum 10
restarts per minute; each of these settings could be left out, which means
that the respective limit does not apply, leaving the possibility to specify an
absolute upper limit on the restarts or to make the restarts work infinitely.</p>
<p>The match statement which forms the bulk of the body is of type <tt class="docutils literal"><span class="pre">Decider</span></tt>,
which is a <tt class="docutils literal"><span class="pre">PartialFunction[Throwable,</span> <span class="pre">Directive]</span></tt>. This
is the piece which maps child failure types to their corresponding directives.</p>
</div>
<div class="section" id="default-supervisor-strategy">
<h2>Default Supervisor Strategy</h2>
<p><tt class="docutils literal"><span class="pre">Escalate</span></tt> is used if the defined strategy doesn&#8217;t cover the exception that was thrown.</p>
<p>When the supervisor strategy is not defined for an actor the following
exceptions are handled by default:</p>
<ul class="simple">
<li><tt class="docutils literal"><span class="pre">ActorInitializationException</span></tt> will stop the failing child actor</li>
<li><tt class="docutils literal"><span class="pre">ActorKilledException</span></tt> will stop the failing child actor</li>
<li><tt class="docutils literal"><span class="pre">Exception</span></tt> will restart the failing child actor</li>
<li>Other types of <tt class="docutils literal"><span class="pre">Throwable</span></tt> will be escalated to parent actor</li>
</ul>
<p>If the exception escalate all the way up to the root guardian it will handle it
in the same way as the default strategy defined above.</p>
</div>
<div class="section" id="test-application">
<h2>Test Application</h2>
<p>The following section shows the effects of the different directives in practice,
wherefor a test setup is needed. First off, we need a suitable supervisor:</p>
<div class="highlight-scala"><div class="highlight"><pre><span class="k">import</span> <span class="nn">akka.actor.Actor</span>

<span class="k">class</span> <span class="nc">Supervisor</span> <span class="k">extends</span> <span class="nc">Actor</span> <span class="o">{</span>
  <span class="k">import</span> <span class="nn">akka.actor.OneForOneStrategy</span>
  <span class="k">import</span> <span class="nn">akka.actor.SupervisorStrategy._</span>
  <span class="k">import</span> <span class="nn">akka.util.duration._</span>

  <span class="k">override</span> <span class="k">val</span> <span class="n">supervisorStrategy</span> <span class="k">=</span> <span class="nc">OneForOneStrategy</span><span class="o">(</span><span class="n">maxNrOfRetries</span> <span class="k">=</span> <span class="mi">10</span><span class="o">,</span> <span class="n">withinTimeRange</span> <span class="k">=</span> <span class="mi">1</span> <span class="n">minute</span><span class="o">)</span> <span class="o">{</span>
    <span class="k">case</span> <span class="n">_:</span> <span class="nc">ArithmeticException</span>      <span class="k">⇒</span> <span class="nc">Resume</span>
    <span class="k">case</span> <span class="n">_:</span> <span class="nc">NullPointerException</span>     <span class="k">⇒</span> <span class="nc">Restart</span>
    <span class="k">case</span> <span class="n">_:</span> <span class="nc">IllegalArgumentException</span> <span class="k">⇒</span> <span class="nc">Stop</span>
    <span class="k">case</span> <span class="n">_:</span> <span class="nc">Exception</span>                <span class="k">⇒</span> <span class="nc">Escalate</span>
  <span class="o">}</span>

  <span class="k">def</span> <span class="n">receive</span> <span class="k">=</span> <span class="o">{</span>
    <span class="k">case</span> <span class="n">p</span><span class="k">:</span> <span class="kt">Props</span> <span class="k">⇒</span> <span class="kt">sender</span> <span class="kt">!</span> <span class="kt">context.actorOf</span><span class="o">(</span><span class="kt">p</span><span class="o">)</span>
  <span class="o">}</span>
<span class="o">}</span>
</pre></div>
</div>
<p>This supervisor will be used to create a child, with which we can experiment:</p>
<div class="highlight-scala"><div class="highlight"><pre><span class="k">import</span> <span class="nn">akka.actor.Actor</span>

<span class="k">class</span> <span class="nc">Child</span> <span class="k">extends</span> <span class="nc">Actor</span> <span class="o">{</span>
  <span class="k">var</span> <span class="n">state</span> <span class="k">=</span> <span class="mi">0</span>
  <span class="k">def</span> <span class="n">receive</span> <span class="k">=</span> <span class="o">{</span>
    <span class="k">case</span> <span class="n">ex</span><span class="k">:</span> <span class="kt">Exception</span> <span class="k">⇒</span> <span class="kt">throw</span> <span class="kt">ex</span>
    <span class="k">case</span> <span class="n">x</span><span class="k">:</span> <span class="kt">Int</span>        <span class="k">⇒</span> <span class="kt">state</span> <span class="o">=</span> <span class="n">x</span>
    <span class="k">case</span> <span class="s">&quot;get&quot;</span>         <span class="k">⇒</span> <span class="n">sender</span> <span class="o">!</span> <span class="n">state</span>
  <span class="o">}</span>
<span class="o">}</span>
</pre></div>
</div>
<p>The test is easier by using the utilities described in <a class="reference internal" href="testing.html#akka-testkit"><em>Testing Actor Systems (Scala)</em></a>,
where <tt class="docutils literal"><span class="pre">AkkaSpec</span></tt> is a convenient mixture of <tt class="docutils literal"><span class="pre">TestKit</span> <span class="pre">with</span> <span class="pre">WordSpec</span> <span class="pre">with</span>
<span class="pre">MustMatchers</span></tt></p>
<div class="highlight-scala"><div class="highlight"><pre><span class="k">import</span> <span class="nn">akka.testkit.</span><span class="o">{</span> <span class="nc">AkkaSpec</span><span class="o">,</span> <span class="nc">ImplicitSender</span><span class="o">,</span> <span class="nc">EventFilter</span> <span class="o">}</span>
<span class="k">import</span> <span class="nn">akka.actor.</span><span class="o">{</span> <span class="nc">ActorRef</span><span class="o">,</span> <span class="nc">Props</span><span class="o">,</span> <span class="nc">Terminated</span> <span class="o">}</span>

<span class="k">class</span> <span class="nc">FaultHandlingDocSpec</span> <span class="k">extends</span> <span class="nc">AkkaSpec</span> <span class="k">with</span> <span class="nc">ImplicitSender</span> <span class="o">{</span>

  <span class="s">&quot;A supervisor&quot;</span> <span class="n">must</span> <span class="o">{</span>

    <span class="s">&quot;apply the chosen strategy for its child&quot;</span> <span class="n">in</span> <span class="o">{</span>
      <span class="c1">// code here</span>
    <span class="o">}</span>
  <span class="o">}</span>
<span class="o">}</span>
</pre></div>
</div>
<p>Let us create actors:</p>
<div class="highlight-scala"><div class="highlight"><pre><span class="k">val</span> <span class="n">supervisor</span> <span class="k">=</span> <span class="n">system</span><span class="o">.</span><span class="n">actorOf</span><span class="o">(</span><span class="nc">Props</span><span class="o">[</span><span class="kt">Supervisor</span><span class="o">],</span> <span class="s">&quot;supervisor&quot;</span><span class="o">)</span>

<span class="n">supervisor</span> <span class="o">!</span> <span class="nc">Props</span><span class="o">[</span><span class="kt">Child</span><span class="o">]</span>
<span class="k">val</span> <span class="n">child</span> <span class="k">=</span> <span class="n">expectMsgType</span><span class="o">[</span><span class="kt">ActorRef</span><span class="o">]</span> <span class="c1">// retrieve answer from TestKit’s testActor</span>
</pre></div>
</div>
<p>The first test shall demonstrate the <tt class="docutils literal"><span class="pre">Resume</span></tt> directive, so we try it out by
setting some non-initial state in the actor and have it fail:</p>
<div class="highlight-scala"><div class="highlight"><pre><span class="n">child</span> <span class="o">!</span> <span class="mi">42</span> <span class="c1">// set state to 42</span>
<span class="n">child</span> <span class="o">!</span> <span class="s">&quot;get&quot;</span>
<span class="n">expectMsg</span><span class="o">(</span><span class="mi">42</span><span class="o">)</span>

<span class="n">child</span> <span class="o">!</span> <span class="k">new</span> <span class="nc">ArithmeticException</span> <span class="c1">// crash it</span>
<span class="n">child</span> <span class="o">!</span> <span class="s">&quot;get&quot;</span>
<span class="n">expectMsg</span><span class="o">(</span><span class="mi">42</span><span class="o">)</span>
</pre></div>
</div>
<p>As you can see the value 42 survives the fault handling directive. Now, if we
change the failure to a more serious <tt class="docutils literal"><span class="pre">NullPointerException</span></tt>, that will no
longer be the case:</p>
<div class="highlight-scala"><div class="highlight"><pre><span class="n">child</span> <span class="o">!</span> <span class="k">new</span> <span class="nc">NullPointerException</span> <span class="c1">// crash it harder</span>
<span class="n">child</span> <span class="o">!</span> <span class="s">&quot;get&quot;</span>
<span class="n">expectMsg</span><span class="o">(</span><span class="mi">0</span><span class="o">)</span>
</pre></div>
</div>
<p>And finally in case of the fatal <tt class="docutils literal"><span class="pre">IllegalArgumentException</span></tt> the child will be
terminated by the supervisor:</p>
<div class="highlight-scala"><div class="highlight"><pre><span class="n">watch</span><span class="o">(</span><span class="n">child</span><span class="o">)</span> <span class="c1">// have testActor watch “child”</span>
<span class="n">child</span> <span class="o">!</span> <span class="k">new</span> <span class="nc">IllegalArgumentException</span> <span class="c1">// break it</span>
<span class="n">expectMsg</span><span class="o">(</span><span class="nc">Terminated</span><span class="o">(</span><span class="n">child</span><span class="o">))</span>
<span class="n">child</span><span class="o">.</span><span class="n">isTerminated</span> <span class="n">must</span> <span class="n">be</span><span class="o">(</span><span class="kc">true</span><span class="o">)</span>
</pre></div>
</div>
<p>Up to now the supervisor was completely unaffected by the child’s failure,
because the directives set did handle it. In case of an <tt class="docutils literal"><span class="pre">Exception</span></tt>, this is not
true anymore and the supervisor escalates the failure.</p>
<div class="highlight-scala"><div class="highlight"><pre><span class="n">supervisor</span> <span class="o">!</span> <span class="nc">Props</span><span class="o">[</span><span class="kt">Child</span><span class="o">]</span> <span class="c1">// create new child</span>
<span class="k">val</span> <span class="n">child2</span> <span class="k">=</span> <span class="n">expectMsgType</span><span class="o">[</span><span class="kt">ActorRef</span><span class="o">]</span>

<span class="n">watch</span><span class="o">(</span><span class="n">child2</span><span class="o">)</span>
<span class="n">child2</span> <span class="o">!</span> <span class="s">&quot;get&quot;</span> <span class="c1">// verify it is alive</span>
<span class="n">expectMsg</span><span class="o">(</span><span class="mi">0</span><span class="o">)</span>

<span class="n">child2</span> <span class="o">!</span> <span class="k">new</span> <span class="nc">Exception</span><span class="o">(</span><span class="s">&quot;CRASH&quot;</span><span class="o">)</span> <span class="c1">// escalate failure</span>
<span class="n">expectMsg</span><span class="o">(</span><span class="nc">Terminated</span><span class="o">(</span><span class="n">child2</span><span class="o">))</span>
</pre></div>
</div>
<p>The supervisor itself is supervised by the top-level actor provided by the
<tt class="xref py py-class docutils literal"><span class="pre">ActorSystem</span></tt>, which has the default policy to restart in case of all
<tt class="docutils literal"><span class="pre">Exception</span></tt> cases (with the notable exceptions of
<tt class="docutils literal"><span class="pre">ActorInitializationException</span></tt> and <tt class="docutils literal"><span class="pre">ActorKilledException</span></tt>). Since the
default directive in case of a restart is to kill all children, we expected our poor
child not to survive this failure.</p>
<p>In case this is not desired (which depends on the use case), we need to use a
different supervisor which overrides this behavior.</p>
<div class="highlight-scala"><div class="highlight"><pre><span class="k">class</span> <span class="nc">Supervisor2</span> <span class="k">extends</span> <span class="nc">Actor</span> <span class="o">{</span>
  <span class="k">import</span> <span class="nn">akka.actor.OneForOneStrategy</span>
  <span class="k">import</span> <span class="nn">akka.actor.SupervisorStrategy._</span>
  <span class="k">import</span> <span class="nn">akka.util.duration._</span>

  <span class="k">override</span> <span class="k">val</span> <span class="n">supervisorStrategy</span> <span class="k">=</span> <span class="nc">OneForOneStrategy</span><span class="o">(</span><span class="n">maxNrOfRetries</span> <span class="k">=</span> <span class="mi">10</span><span class="o">,</span> <span class="n">withinTimeRange</span> <span class="k">=</span> <span class="mi">1</span> <span class="n">minute</span><span class="o">)</span> <span class="o">{</span>
    <span class="k">case</span> <span class="n">_:</span> <span class="nc">ArithmeticException</span>      <span class="k">⇒</span> <span class="nc">Resume</span>
    <span class="k">case</span> <span class="n">_:</span> <span class="nc">NullPointerException</span>     <span class="k">⇒</span> <span class="nc">Restart</span>
    <span class="k">case</span> <span class="n">_:</span> <span class="nc">IllegalArgumentException</span> <span class="k">⇒</span> <span class="nc">Stop</span>
    <span class="k">case</span> <span class="n">_:</span> <span class="nc">Exception</span>                <span class="k">⇒</span> <span class="nc">Escalate</span>
  <span class="o">}</span>

  <span class="k">def</span> <span class="n">receive</span> <span class="k">=</span> <span class="o">{</span>
    <span class="k">case</span> <span class="n">p</span><span class="k">:</span> <span class="kt">Props</span> <span class="k">⇒</span> <span class="kt">sender</span> <span class="kt">!</span> <span class="kt">context.actorOf</span><span class="o">(</span><span class="kt">p</span><span class="o">)</span>
  <span class="o">}</span>
  <span class="c1">// override default to kill all children during restart</span>
  <span class="k">override</span> <span class="k">def</span> <span class="n">preRestart</span><span class="o">(</span><span class="n">cause</span><span class="k">:</span> <span class="kt">Throwable</span><span class="o">,</span> <span class="n">msg</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">Any</span><span class="o">])</span> <span class="o">{}</span>
<span class="o">}</span>
</pre></div>
</div>
<p>With this parent, the child survives the escalated restart, as demonstrated in
the last test:</p>
<div class="highlight-scala"><div class="highlight"><pre><span class="k">val</span> <span class="n">supervisor2</span> <span class="k">=</span> <span class="n">system</span><span class="o">.</span><span class="n">actorOf</span><span class="o">(</span><span class="nc">Props</span><span class="o">[</span><span class="kt">Supervisor2</span><span class="o">],</span> <span class="s">&quot;supervisor2&quot;</span><span class="o">)</span>

<span class="n">supervisor2</span> <span class="o">!</span> <span class="nc">Props</span><span class="o">[</span><span class="kt">Child</span><span class="o">]</span>
<span class="k">val</span> <span class="n">child3</span> <span class="k">=</span> <span class="n">expectMsgType</span><span class="o">[</span><span class="kt">ActorRef</span><span class="o">]</span>

<span class="n">child3</span> <span class="o">!</span> <span class="mi">23</span>
<span class="n">child3</span> <span class="o">!</span> <span class="s">&quot;get&quot;</span>
<span class="n">expectMsg</span><span class="o">(</span><span class="mi">23</span><span class="o">)</span>

<span class="n">child3</span> <span class="o">!</span> <span class="k">new</span> <span class="nc">Exception</span><span class="o">(</span><span class="s">&quot;CRASH&quot;</span><span class="o">)</span>
<span class="n">child3</span> <span class="o">!</span> <span class="s">&quot;get&quot;</span>
<span class="n">expectMsg</span><span class="o">(</span><span class="mi">0</span><span class="o">)</span>
</pre></div>
</div>
</div>
</div>


          </div>
          <div class="span3"><p class="contents-title">Contents</p>
              <div id="scroller-anchor">
                <div id="scroller">
                  <div id="toc"></div>
                </div>
              </div></div>
        </div>
      </div>
    </div>
  </div>
  <div class="footer">
  <div class="container">
    <ul>
      <li><h5>Akka</h5></li>
      <li><a href="http://akka.io/docs">Documentation</a></li>
      <li><a href="http://akka.io/downloads">Downloads</a></li>
    </ul>
    <ul>
      <li><h5>Contribute</h5></li>
      <li><a href="http://github.com/akka/akka">Source Code</a></li>
      <li><a href="http://groups.google.com/group/akka-user">Mailing List</a></li>      
      <li><a href="http://www.assembla.com/spaces/akka/tickets">Report a Bug</a></li>      
    </ul>
    <ul>
      <li><h5>Company</h5></li>
      <li><a href="http://typesafe.com/products/typesafe-subscription">Commercial Support</a></li>
      <li><a href="http://akka.io/team">Team</a></li>
      <li><a href="mailto:info@typesafe.com">Contact</a></li>
    </ul>
    <ul>
      <li><img src="../_static/watermark.png" align="center"/></li>
    </ul>
  </div>
  <div class="container copyright">
    <p style="float: left;">
      © 2012 <a href="http://typesafe.com/">Typesafe Inc.</a> <span class="license">Akka is Open Source and available under the Apache 2 License.</span>
    </p>
    <p style="float: right; font-size: 12px;">
      Last updated: Apr 13, 2012
    </p>          
  </div>
</div>
<script type="text/javascript">
  $('#toc').toc();
</script>
  

  </body>
</html>